Update with the new mainstream structure - #5

* option1

* also debugging the judge

* also debugging the judge

* debug

* eval tracker fix 1

* likely fix for the GSM+ issue

* stringify model judge + change max_length to what's actually passed instead of setting a bunch of overwrites

* more memory for flow judge

…several combinations (huggingface#1017)

* fix

* added a warning message

* fix unit tests

* fix unit tests 2

* mini fix

* minifix

* test

* update new metrics name

* updated var names

…huggingface#828)

Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com>

* homogeneize k and n in parametrizable metrics

* updated aime, last metric fixs

* fix

* restore rm import

* restore

* update doc

* gpqa fix

* pass at

* recall

* test

Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>

* use inspect-ai to evaluate aime25 and gsm8k

* revert file

* working for 3 tasks

* parallel evals of tasks

* adds gpqa diamond to inspect

* move tasks to individual files

* move tasks to individual files

* enable extended tasks as well

* run precomit hook

* fix mkqa

* chaange extended suite to lighteval

* chaange extended suite to lighteval

* add metdata to tasks

* add metdata to tasks

* remove license notice and put docstring on top of file

* homogenize tags

* add docstring for all multilingual tasks

* add docstring for all multilingual tasks

* add name and dataset to metadata

* use TASKS_TABLE for multilingual tasks

* use TASKS_TABLE for default tasks

* use TASKS_TABLE for default tasks

* loads all tasks correclty

* move community tasks to default tasks and update doc

* move community tasks to default tasks and update doc

* revert uneeded changes

* fix doc build

* fix doc build

* remove custom tasks and let user decide if loading multilingual tasks

* load-tasks multilingual fix

* update doc

* remove uneeded file

* update readme

* update readme

* update readme

* fix test

* add back the custom tasks

* add back the custom tasks

* fix tasks

* fix tasks

* fix tasks

* fix tests

* fix tests

adds inspect-ai as backend for lighteval! Offloading backend implementation and maintenance

- this allows for:
- better logs
- better paralelixzation
- easier to add tasks

tasks compatible with inspect ai (at term all the tasks will be compatible):

- gpqa (fewshot compatible)
- ifeval
- hle
- gsm8k (fewshot compatible)
- agieval
- aime24,25

### run llama3.1-8b using all providers on `hf-inference-providers` on `gpqa`, `agieval` and `aime25`:

```
lighteval eval hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:cerebras \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:featherless-ai \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:fireworks-ai \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:novita \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:nebius \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:sambanova \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:scaleway \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:nscale \
"lighteval|gpqa|0,lighteval|agieval|0,lighteval|aime25|0" \
max-connections 50 --timeout 30  --retry-on-error 1 --max-retries 5 --epochs 1 --max-samples 1
```

result:

```
|                                Model                                 |agieval|aime25|gpqa|
|----------------------------------------------------------------------|------:|-----:|---:|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:cerebras      |   0.53|     0|0.33|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:featherless-ai|   0.71|     1|0.75|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:fireworks-ai  |   0.71|     0|0.25|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:nebius        |   0.53|     0|0.20|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:novita        |   0.65|     0|0.75|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:sambanova     |   0.71|     0|0.25|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:scaleway      |   0.35|     0|0.25|
```


### compare few shots diff on gsm8k

```
lighteval eval hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:cerebras \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:featherless-ai \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:fireworks-ai \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:novita \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:nebius \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:sambanova \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:scaleway \
hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:nscale \
"lighteval|gsm8k|0,lighteval|gsm8k|3" \
max-connections 50 --timeout 30  --retry-on-error 1 --max-retries 5 --epochs 1 --max-samples 1
```

```
|                                Model                                 |gsm8k|gsm8k_3_shots|
|----------------------------------------------------------------------|----:|------------:|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:cerebras      |  0.6|          0.7|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:featherless-ai|  0.7|          0.7|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:fireworks-ai  |  0.7|          0.8|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:nebius        |  0.6|          0.7|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:novita        |  0.5|          0.7|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:sambanova     |  0.7|          0.7|
|hf-inference-providers/meta-llama/Llama-3.1-8B-Instruct:scaleway      |  0.4|          0.8|
```

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* adds mmlu-pro

* adds mmlu-pro

* add mmlu-pro with inspectai

* adds mmlu-pro

* adds mmlu-pro

* add mmlu-pro with inspectai

* fix reasoning effrot

…1034)

…#992)

* fix

* revert uneeded changes

---------

Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>

* run all hf-providers

* add example

* remove uneeded params

* remove suites and make fewshot optional

* fix docs to remove suites and fewshots

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>

Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>

…uggingface#1051)

* remove suites and make fewshot optional

* fix docs to remove suites and fewshots

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* Remove suite argument iin task config

* Remove suite argument iin task config

* fix try to cache functool.partial function

* fix styling

…gface#1052)

* add a task dump in registry for better documentation of tasks

* Update src/lighteval/tasks/registry.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/lighteval/tasks/registry.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/lighteval/tasks/registry.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix

* remove

* fix aimo

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

set(1,2,3) -> {1,2,3}

Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
Co-authored-by: Francesco Bertolotti <francesco.bertolotti@gmail.com>

Co-authored-by: Kangda Wei <kangdawei@Kangdas-MacBook-Pro.local>

even though vllm produces openai compatible endpoint, to make work you have to use provider as hosted_vllm and use a hosted_vllm prefix prior to model name

moves all the prompts from `default_prompts.py` to their respective task file

- score the query as the continuation when choices is None (the_pile/wikitext)
- shift logits by one so tokens line up with their predictions
- stop the sample cache dropping PERPLEXITY results
- add a rolling-perplexity test and check the cached wrapper returns all docs

litellm.completion expects an int, not a (N,) tuple.

Current RAG-style tasks need the row-specific retrieved context to
live in the system role, not prepended to the user query. Opt-in
flag keeps all existing tasks unchanged.

…ge LLM (to avoid some memory errors)

…ntually called)

squad_v2 was filtering out questions with no answer, which is
exactly the half of the dataset that tests refusal behavior.
Replace the filter with an explicit "unanswerable" choice.

…options, not all the possible ones. Also increase generation_size from 100 to 1024 (for thinking models)

The generator had been narrowed to MCFFormulation + the ALL label only,
which dropped the _cf/_hybrid variants and the CA/CS/UNK labels. Restore
the full formulation list and sensitivity labels.

…dict=False to get token ids, not a BatchEncoding)

…uggingface#1264)

…huggingface#1067 regression)

…text' setting

…ggingface#1271)

Under an accelerate data-parallel launch every rank holds the full gathered
results and wrote the same parquet cache file concurrently, corrupting it and
making subsequent loads fail. Write the cache only on the main process and add
a barrier so the other ranks wait for that write before reading. Add a
regression test.

Fixes huggingface#1102

…match of the instruction)

…#1326)

* Store provider credentials as SecretStr in model configs

LiteLLMModelConfig.api_key and TGIModelConfig.inference_server_auth were
plain str fields, which meant they were retained in plaintext wherever a
model config gets serialized (e.g. EvaluationTracker.results). Switch
both to pydantic SecretStr, which masks the value in reprs and default
serialization, and additionally exclude them explicitly when building
the results dict as a second layer.

The real value is still unwrapped via get_secret_value() at the specific
call sites that need it for the actual outgoing request. Added
regression tests asserting the credential never appears in the
serialized results dict.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Store JudgeLM credentials as SecretStr

JudgeLM.api_key was a plain str consumed directly by several backend
clients (OpenAI, AsyncInferenceClient, litellm). Wrap it in SecretStr on
assignment and unwrap via get_secret_value() at each usage site, for
consistency with the other model configs and to remove any reliance on
incidental string formatting to keep it out of logs or serialized
output.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Fix pre-existing ruff format drift in README and docs

Unrelated cleanup so CI's Quality check is green on this branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Nathan Habib <nathan_habib@Mac.lan>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Bring in 19 upstream commits (up to 932e1f2), including Swiss legal NLP
tasks + LEXam, SecretStr credential storage, rolling-perplexity and
bits-per-byte fixes, and multilingual task loading in vLLM.

Conflicts resolved in:
- metrics/__init__.py: kept the fork's permissive batched-output handling
  (extracts all submetric keys) to support the fork's custom batched metrics.
- metrics/utils/llm_as_judge.py: kept the fork's max_tokens int fix (not the
  upstream (max_new_tokens,) tuple) and took upstream's SecretStr api_key /
  base_url plumbing.